Rear-view mirror simulation

ABSTRACT

A method displays information graphically on an image captured by an optical system used to simulate a rear-view mirror system of a vehicle. The method includes capturing the image and identifying an object in the image. The method the assigns a priority level to the object based on a predetermined criterion. The object is altered based on its priority level to create an altered object. An altered object may have a changed color, a colored halo surrounding it, or a colored object inside it. Other possible ways to alter the way the object looks are possible. The altered object is displayed in the image in place of the object to alert a vehicle operator of its presence and the priority level associated therewith.

This patent application is a continuation-in-part of U.S. patentapplication Ser. No. 15/691,332, which is a continuation-in-part of U.S.patent application Ser. No. 15/602,068, which itself is a continuationof U.S. patent application Ser. No. 14/968,132, which is a continuationof U.S. patent application Ser. No. 13/090,127. Furthermore, this patentapplication claims the priority of U.S. Ser. No. 15/287,554, which ishereby incorporated herein by reference. The invention is based on thepriority patent application U.S. Ser. No. 13/090,127 and U.S. Ser. No.15/287,554 which are hereby incorporated by reference.

BACKGROUND ART

Field of the Invention

The invention relates to an exterior mirror simulation with image datarecording and a display of the recorded and improved data for the driverof a vehicle.

The display on a display device shows the data in a way favored by thedriver and/or vehicle manufacturer.

Description of the Related Art

Several solutions for recording image data and its display for thedriver of a vehicle are known in the prior art. The image recording isdone by one or several cameras installed in the vehicle. The differentassistance systems process the data from the captured image in verydifferent ways.

In US patent application having publication number 2008/0159594, asystem is known which records images from the surroundings of thevehicle with a fish-eye lens. Image data is recorded with greatdistortion through this wide-angle lens. The image data recorded by thecamera pixels are rectified block by block. The display of the image isdone with the rectified image data, since an image of the surroundingsof the vehicle is required.

A blind spot indicator is disclosed in U.S. Pat. No. 8,779,911 B2, whichis adjacent to a second mirror surface of a rear-view device, aso-called spotter area, used to observe objects located in a blind spotof the vehicle.

A camera for assisting reversing is known in DE 102008031784. Thedistorted camera image is edited and rectified, which leads to anundistorted image. This is then further processed, in order to optimizethe perspective for reversing.

Currently the camera images are presented to the user in a mostlyrealistic way, with enhancements in image quality. Additionalinformation, if shown at all, is presented as a layer on top of theoriginal object and therewith appears more like a foreign object.

Drawing artificial lines is common in parking assist camera systems tovisualize the lane in dependence of the steering-wheel position. Amethod for visualizing a vicinity of a motor vehicle is for exampledescribed in US 2013/0201335 A1, with said method comprising the stepsof: recording the vicinity of the motor vehicle with a camera;displaying the camera image on a display; and providing another detectedroad user in the camera image with a framing of a defined color, whereinthe color is a function of a speed of the detected road user relative tothe motor vehicle and is indicative of whether or not the speed of thedetected road user is critical.

SUMMARY OF THE INVENTION

The object of the invention is to alter the images captured by a camerasystem, for example used for electronic mirror replacement, to providethe user with further information. In particular it is the object toblend said further information into natural images in order tofacilitate the perception of a current or future traffic situation bythe driver of a vehicle.

In addition, the present invention relates to creating a display of acamera image, which corresponds to the familiar image in a rear-viewmirror. The distortions of the image caused by the different mirrorglasses are provided for the driver in the usual manner.

The present invention also relates to image rectification for a vehicle,which comprises a display device, in order to show modified images andan imaging means and/or screen for receiving the recorded images, whichhave been improved by image rectification. Furthermore, the systemcomprises image rectification in communication with the display deviceand the imaging means and/or screen, so that pixels, which are locatedin the recorded images, are improved by reorientation or repositioningof the pixels from a first position to a second position using atransmission or transfer process.

Furthermore, the invention relates to a rear-view image improvementsystem for a vehicle, which includes a display device for showingmodified images, which have been improved by the image improvementsystem, and an imaging means for receiving recorded images, which havebeen improved by the image improvement system. The system also comprisesan image improvement module in connection with the display device andthe imaging means, and indeed in such a way that pixels, which arelocated in the recorded images, are grouped and spread out, in order toform at least one region of interest, in which reference is made to thepixels from a base plane in the recorded image, in order to form themodified images.

Furthermore, the invention relates to a vehicle comprising displayingdevices, processing devices and sensors such as cameras.

The object of the invention is to also provide an object detection andclassification system with image feature descriptors derived fromperiodic descriptor functions.

An object detection and classification system analyzes images capturedby an image sensor like a camera, a radar sensor and/or ultrasonicsensor, for a hazard detection and information system, such as on avehicle. Extracting circuitry is configured to extract at least onefeature value from one or more keypoints in an image captured by animage sensor of the environment surrounding a vehicle. A new imagefeature descriptor is derived from a periodic descriptor function, whichdepends on the distance between at least one of the keypoints and achosen query point in complex space and depends on a feature value of atleast one of the keypoints in the image.

Query point evaluation circuitry is configured to sample the periodicdescriptor function for a chosen query point in the image from theenvironment surrounding the vehicle to produce a sample value. Thesample value for a query point may be evaluated to determine whether thequery point is the center of an object or evaluated to determine whattype of object the query point is a part of.

If the evaluated query point satisfies a potential hazard condition,such as if the object is classified as a vulnerable road user or objectposing a collision threat, a signal bus is configured to transmit asignal to alert the operator of the vehicle to the object. Additionally,or alternatively, the signal bus may transmit a signal to a controlapparatus of the vehicle to alter the vehicle's speed and/or directionto avoid collision with the object.

The object detection and classification system disclosed herein may beused in the area of transportation for identifying and classifyingobjects encountered in the environment surrounding a vehicle, such as onthe road, rail, water, air, etc., and alerting the operator of thevehicle or autonomously taking control of the vehicle if the systemdetermines the encountered object poses a hazard, such as a risk ofcollision or danger to the vehicle or to other vehicles or persons inthe area.

Another aspect of this invention is a rearview device and illuminationmeans comprising different functions.

BRIEF DESCRIPTION OF THE DRAWINGS

Advantages of the invention will be readily appreciated as the samebecomes better understood by reference to the following detaileddescription when considered in connection with the accompanyingdrawings, wherein:

FIG. 1 shows an exemplary exterior mirror.

FIG. 2 shows examples of different mirror types.

FIG. 3a-3k show exemplarily different forms of color scales.

FIG. 3m illustrates one embodiment of the visualization ofvehicle-specific velocity and safeness through colored contours aroundobjects.

FIG. 4 shows a camera installation.

FIG. 5 shows an exemplary vehicle.

FIG. 6 shows a display in the vehicle.

FIG. 7 shows the process of image capture.

FIG. 8 shows an alternative process.

FIG. 9 shows distorted and rectified pixel areas.

FIG. 10 shows the basic model of an image capturing device.

FIG. 11 shows the relationship between output image and surface model.

FIG. 12 shows a method to produce a single image area from multipleimages.

FIG. 13 shows a first method to match the edges of two or more borderingimage areas.

FIG. 14 shows the adaption of the first method to multiple image areas,when only independent edges exist.

FIG. 15 shows a part of a second method to match the edges of two ormore bordering image areas.

FIG. 16 shows another part of the second method to match the edges oftwo or more bordering image areas.

FIG. 17 illustrates a rear view of a vehicle with an object detectionand classification system;

FIG. 18 illustrates a schematic of an image capture with a query pointand a plurality of keypoints;

FIG. 19 illustrates a block diagram of a system that may be useful inimplementing the implementations disclosed herein;

FIG. 20 illustrates example operations for detecting and classifying anobject and transmitting a signal to an alert system and/or to a vehiclecontrol system; and

FIG. 21 discloses a block diagram of an example processor systemsuitable for implementing one or more aspects of an object detection andclassification system with Fourier fans.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exterior mirror 1, which comprises a mirror head 2,which is connected to a vehicle by a mirror base or bracket 3. Themirror head 2 forms the and opening and/or slot for a mirror glass 4.

The size of the mirror glass is determined by the mounting on thevehicle, as well as by the different legal regulations about theavailable field of view. In this process, different glass types fordifferent regions have developed. In the USA, a flat plane glass is usedon the driver side.

A mirror glass 4 with a curvature radius is shown in FIG. 2. The glassin FIG. 2 can be used in mirror assemblies on the passenger side of thevehicle and on the driver side of the vehicle in countries other thanthe USA. Convex mirror glasses as well as glass with an aspherical partare used in addition to convex glass.

The driver of a vehicle is used to the display of each type of exteriormirror, and therefore can deduce for himself the warning informationwhich he needs to steer the vehicle through the traffic.

Alternatively, or in addition the mirror and/or the display device cancomprise an additional display or part of a display and/or illuminationmeans within and/or surrounding at least parts of the mirror and/or thedisplay device to convey additional information to the driver or otherpersons looking at the mirror and/or the display device. This additionalinformation can enhance the perception of the current situation, forexample the situation typically depicted in an exterior mirror, byproviding detailed information, for example about the surroundings ofthe vehicle, for example a blind spot indicator, a traffic jamindicator, a weather indicator, an object distance indicator and/orabout parameters of the vehicle, for example the speed of the vehicle,the driving direction of the vehicle, the vehicle mode, a warningindicator. For this a graphical representation can be used, for examplea color scale.

The color scale can take various forms, comprising for example amultitude of elements, for example arranged vertically as shown forstripes in FIG. 3a , arranged in a circle as shown for stripes in FIG.3b , arranged in a half-circle as shown for stripes in FIG. 3c ,arranged in a triangle shape as shown for stripes in FIG. 3d , arrangedin a rectangular shape as shown for stripes in FIG. 3e . The shape ofthe elements can also vary and is not limited to the shown stripes,comprising for example triangles, circles, squares, 2D and/or 3Drepresentations of 3D objects, for example cubes, boxes, pyramids andmany more.

The scale can also comprise just a single element, becoming smaller orlarger and/or changing colors. Preferably the single element comprises acontinuous changing color scale, of which several possible embodimentsare shown in FIGS. 3f -3 k.

At the same time, a number representation of the parameter and/or theparameter range can be displayed next to the scale to increase theperception by the driver. The orientation of the scale can be choseneither horizontal, vertical and/or at any angle in between.

The size, shape color and volume of the graphical representation canalso change with the at least one parameter of the vehicle and/orenvironment, such that for example a single or multiple elements fadeaway, disappear and/or appear. The arrows shown in the FIGS. 3a-3kindicates exemplarily the direction of such possible changes.

The graphical representations, for example those shown in FIGS. 3a-3k ,can also be used to be placed adjacent to and/or surrounding and/or inclose proximity to a present spotter area and/or a mirror and/or part ofa mirror of a rear-view device, irrespective if an actual mirror or amirror replacement, such as a display, is used.

In all embodiments, the changes can also be carried out on multipleparts and in multiple directions, sequentially or at the same time

For a human observer, the absolute and relative speed of other vehiclesare difficult to estimate, when those other vehicles are moving directlytowards or away from the direction of observation. This makes itdifficult to evaluate the speed of vehicles in front, of oncomingtraffic and of vehicles in behind, regardless whether they are seendirectly, through a mirror, or through a camera monitor system. Althoughthe relative speed between the driver's vehicle and its surroundingvehicles is crucial to the evaluation of the safeness of an upcomingmaneuver—such as passing or lane change—humans are not good in quicklyand accurately evaluating the safeness of those maneuvers. As anexample, drivers who observe rearward traffic on a highway mightunderestimate the relative speed of an approaching car and falselyconsider it safe to change the lane. This situation often leads tocollisions.

Different sensor systems and electronic systems with different technicalapproaches already allow the estimation of other vehicles' velocitiesand also of the safeness of potential maneuvers. This can already befound in blind spot warning systems. Combining the information gained bythe use of different sensor systems allows for a more reliable andefficient determination of important parameters, such as velocity,distance and the like.

One of the advantages of the present invention is to visually presentinformation, especially on the velocity and/or safeness on a digitaldisplay as exemplarily depicted in FIG. 3m . Vehicles 601, 602 areobjects in an image that is captured for displaying information to adriver of the vehicle. These objects 601, 602 are altered so as to drawthe driver's attention to them. In one embodiment, the objects 601, 602are altered by adding a color contour 603, 604, preferably with atransparency gradient outwardly, about the periphery of the objects 601,602 to create a corona or glow effect. Information can be encodedthrough color, contour thickness, transparency, position and the dynamicvariation thereof. For example, a green, thin and light contour 603could stand for a safe situation, whereas a red, thick and dense contour604 signs a potential risk. The warning can also be emphasized by forexample a pulsating thickness and brightness of the contour. The resultis an intuitive, quickly to interpret information presentation that candraw the attention of a driver to a potential dangerous situation;otherwise the presentation can stay more or less in the background inorder to avoid any distraction, e.g. by smoothly fusing with theoriginal image.

The invention also proposes an adaptation on night and day, or moregenerally on light conditions, regarding color, brightness, thickness,transparency contrast, position and/or any other possible characteristicfeature. The present invention also proposes an adaption based onweather or more generally on outside conditions such as rainfall,snowfall, fog and/or road and/or surface conditions such as covered withice, snow, water and/or any kind of dirt. Earth and rock slides or othermoving environmental and potentially dangerous situations can likewisebe presented to the driver.

In night vision systems, it is known that certain objects such asanimals or pedestrians appear in an emphasized way due to thecharacteristic difference in light emission profile in contrast to thesurrounding area.

The invention also alters objects other than vehicles that also may posea hazard to either the object or the vehicle. For example, animals orpedestrians may be determined to be in close enough proximity to thevehicle to warrant being altered by adding a halo or corona. Theproperties of these halos or contours can change based on the situationand/or speed. The camera system of the invention is utilized to presentadditional information in an intuitive, yet subtle way. By utilizing adigital display to provide crucial additional information that aclassical mirror cannot show, the acceptance and appeal of electronicmirror replacements is increased. This will allow a vehicle operator tobe warned of the objects presence and a priority level associated withthe object.

The altered objects are dynamic in that the altered image will change asthe method changes the priority of the object due the relative position,acceleration or speed of the object with respect to the vehicle. Inaddition, external conditions may also change the priority of an alteredobject. Ambient light, whether it is natural or artificial, temperature,precipitation and road conditions, taking snow, ice, fog, dirt and thelike as well as curvatures and slopes into consideration, may all changethe status of an altered object. By way of example, an object that isquickly approaching on drive pavement is not as serious as a quicklyapproaching object on a snow-covered pavement. As stated above, changesin the altered image may include changes in color, brightness,transparency of the halo 603, 604. In addition, the halo 603, 604 mayblink or pulsate to try and attract the attention of the driver.

In an alternative embodiment, the altered object within the image mayinclude an added color within a periphery of the object so that it doesnot have a halo effect surrounding the periphery of the object. Eitherthe entire object within its periphery will be colored or a coloredgeometric shape, such as a circle or a triangle, may be added within theobject to create the altered object.

In one embodiment, there is a single priority level wherein each objectthat reaches that priority level status with be altered in the samemanner. In an alternative embodiment, there is a plurality of prioritylevels, each relating to a separate degree of alter. The objects withinthe image will move between the various priority levels depending ontheir respective status and they will be altered accordingly. Prioritylevels may also differ between animate objects and inanimate objects.

Apart from the here presented solution, there is no known approach topresent vehicle-specific speeds and safeness in a way, that mergesnon-obtrusively with the original image and therewith doesn't appeartechnical, but is easy and intuitively understood.

Exterior mirrors contribute to the overall wind resistance of thevehicle. The aerodynamics of a vehicle are influenced by the exteriormirror. Therefore, it is sensible to replace it with the camera systemthat provides the same field of view while reducing the adverse effecton aerodynamics, and so to minimize the total CO2 emissions of thevehicle, by reducing the turbulent flows around the vehicle, andcreating a predominantly laminar flow.

FIG. 4 shows a possible installation of a rear-view assembly, generallyindicated at 10 in a vehicle. The optical sensor 6, of which only theoptic lens can be seen in the figure, is enclosed in a housing 7. Thehousing 7 is tightly mounted to a vehicle 8, best seen in FIG. 5. Thehousing 7 has a form which is streamlined on the vehicle 8. The opticalsensor itself is installed in the housing 7, and has a watertight sealagainst weather effects, as well as against the influence of washingprocesses with detergents, solvents and high-pressure cleaners.

The housing 7 comprises an opening, through which the camera cabling isled. In this process, the connection of the camera to the electricsystem of the vehicle 8 is done by any bus system or a separate cableconnection. FIG. 5 shows as an example the attachment position of asensor in the housing 7 on the vehicle 8. The camera position istherefore to be chosen in a way that fulfils the legally required fieldof view. The position can therefore be on the front mudguard, on themirror triangle or on the edge of the vehicle roof 8 a. Through theapplication of a wide-angle lens it is possible that the field of viewof the sensor will be larger than through a conventional mirror.

A display device 20, which can be seen by the driver 9, is mounted intoa vehicle 8. The picture from the camera is transmitted to the displaydevice 20. In one embodiment, the display device 20 is mounted to anA-pillar 21 of the motor vehicle 8.

FIG. 6 shows an exemplary embodiment of the present invention 10 with adisplay device 20, which is provided in the vehicle cab or vehicleinterior for observation or viewing by the driver 9. The rear-viewassembly 10 delivers real-time wide-angle video images to the driver 9that are captured and converted to electrical signals by the opticalsensor 6. The optical sensor 6 is, for example, a sensor technology witha Charge-Coupled Device (‘CCD’) or a Complementary Metal OxideSemiconductor (‘CMOS’), for recording continuous real-time images. InFIG. 6, the display device 20 is attached to the A-pillar 21, so thatthe familiar look in the rear-view mirror is led to a position which issimilar to the familiar position of the exterior mirror used up to now.

In the event of mounting on the A-pillar 21 being difficult due to theairbag safety system, a position on the dashboard 22 near to the mirrortriangle or the A-pillar 21 is also an option. The display device showsthe real-time images of camera 6, as they are recorded in this exampleby a camera in the exterior mirror.

The invention is not dependent on whether the exterior mirror iscompletely replaced, or, as it is shown in FIG. 6, it is still availableas additional information. The optical sensor 6 can look through asemitransparent mirror glass.

The field of view recorded by an optical sensor 6 is processed andimproved in an image rectification module, which is associated with therear-view assembly 10, according to the control process shown in FIG. 7.The image rectification module uses a part of the vehicle 8 as areference (e.g. a part of the vehicle contour) when it modifies thecontinuous images, which are transmitted to the display device 20 asvideo data. The display device 20 can be a monitor, a liquid crystaldisplay device or a TFT display, or LCD, a navigation screen or otherknown video display devices, which in the present invention permit thedriver 9 to see the area near to the vehicle 8. The application of OLED,holographic or laser projection displays, which are adapted to thecontour of the dashboard or the A-pillar 21, are also useful.

The image rectification occurs onboard the vehicle 8, and comprisesprocessing capacities, which are carried out by a computation unit, suchas, for example, a digital signal processor or DSP, a field programmablegate array (“FPGA”), microprocessors or circuits specific to use, orapplication specific integrated circuits (‘ASIC’), or a combinationthereof, which show programmability, for example, by a computer-readablemedium such as, for example, software or hardware, which is recorded ina microprocessor, including Read Only Memory (‘ROM’), or as binary imagedata, which can be programmed by a user. The image rectification can beformed integrally with the imaging means 20 or the display device 14, orcan be positioned away in communication (wired or wireless) with boththe imaging means as well as the display device.

The initiation or starting up of the image rectification occurs when thedriver starts the vehicle. At least one display device 20 displayscontinuous images from the side of the vehicle, and transmits thecontinuous images to the image rectification device. The imagerectification device modifies the continuous images and transmits theimproved images by video data to the display device 20, in order to helpthe driver.

The individual steps of image rectification as well as image distortionare shown in FIG. 7. In this process, the invention distorts the imageof the wide-angle camera and applies post-distortion to this image, inorder to give this image, the same view as that of the desired mirrorglass.

The first step is the recording of the image. In a second step, the typeof distortion, to which the image is subjected, is determined.

In a further step, the algorithm is selected, which is adapted to thepresent distortion. An example is explained in DE 102008031784.

An optical distortion correction is an improving function, which isapplied to the continuous images. The optical distortion correctionfacilitates the removal of a perspective effect and a visual distortion,which is caused by a wide-angle lens used in the camera 6. The opticaldistortion correction uses a mathematical model of the distortion, inorder to determine the correct position of the pixels, which arerecorded in the continuous images. The mathematical position alsocorrects the pixel position of the continuous images, as a result of thedifferences between the width and height of a pixel unit due to theaspect or side ratio, which is created by the wide-angle lens.

For certain lenses, which are used by the camera 6, the distortionco-efficient values k1 and k2 can be predetermined, in order to help ineliminating the barrel distortion, which is created by the use of awide-angle lens. The distortion co-efficient values are used for thereal-time correction of the continuous images.

The distortion co-efficient values k1 and k2 can be further adjusted orcoordinated by using an image, which is recorded in the continuousimages, which shows the known straight line, for example, the lanemarkings on a road. According to this aspect of the present invention,the distortion center is registered by analysis of the recordedcontinuous images in the search for the straightest horizontal andvertical lines, whereby the center is situated where the two linesintersect. The recorded image can then be corrected with varied orfine-tuned distortion co-efficient values k1 and k2 in a trial and errorprocess. If, for example, the lines on one side of the image are “barreldistorted” (“barreled”) and lines on the other side of the image are“pin cushion distorted” (“pin-cushioned”), then the center offset mustmove in the direction of the pin-cushioned side. If a value is found,which sufficiently corrects the distortion, then the values for thedistortion center 42 and the distortion coefficient values k1 and k2 canbe used in the mathematical model of optical distortion correction.

As a result of the rectification stage at 63, a low-error image is givenat 64, which can be shown on the display device 20. The image obtainedafter rectification corresponds to the image of a plane mirror, wherebythe simulated mirror surface would be larger than the usual mirrorsurface. If such a plane mirror is simulated, the further steps areeliminated and the data is displayed directly on the display accordingto FIG. 8. The image of a plane mirror is defined by a selection ofpixels of the optical sensor. In this way, as shown in FIG. 9, only thepixels in the middle of the optical sensor are chosen. In order tosimulate the plane mirror in a larger approximation on the hardwaremirror, data must be cut, and the section is limited to a section in themiddle of the image.

The operator which is applied to the pixels in order to achieve thedesired image is determined in the next step 64. For example, thealgorithm is selected in order to again distort the low-error image aswould be shown in mirror glass with an aspheric curve, for example.Therefore, the pixel values must be moved in a certain area in order toobtain the impression of curved mirror glass.

In the next step 65, the post-distortion of the present image is carriedout. For example, a plane mirror with a convex additional mirror ischosen. For this purpose, a defined number of pixels is chosen for thedisplay of the plane mirror surface. In FIG. 9 it is area G which showsplane surfaces in the middle of the optical sensor. For the display ofinformation from the convex mirror, all pixels of the sensor must beused, both area G as well as H, in order to provide data to thewide-angle representation of the image, which is situated in a definedarea of the display.

The information from all pixels is subject to a transformation, and theimage of all pixels is distorted and shown on a small area of thedisplay. In this process, information is collated by suitable operatorsin order to optimally display the image on a lower number of displaypixels.

The base plane is a plane surface comprising pixels from the capturedimages. The vision sphere and the cylinder are two other types ofsurfaces, the captured images can be projected onto to build updifferent parts of the displayed output image.

In this way one or more captured images of one or more optical sensorsare displayed to a drive in a joint output image shown on the displaydevice. In this way, although differently distorted areas are existingwithin the shown output image, a homogenous image is presented to thedriver. Especially the areas within the image can have differentwide-angle distortions. In this way, it is possible to present an outputimage already known for the driver from rear view systems comprising acombination of different types of mirrors, like a combination of amirror with a plane glass surface fading to a mirror having a curvedsurface, like an aspheric curvature. Such mirrors have a wide-angle areaby which a picture is generated that has a view angle that is increasedin comparison to the view angle of the human eye, i.e. is distorted. Inthe post distortion step a first sector of the image area having a firstdistortion that is located adjacent to another image area having adifferent second distortion compared to the first distortion is modifiedin the following way. In the first sector, the first area is homogenizedwith the adjacent second area. For this purpose, pixels that are locatedwithin captured images are grouped and separated to define a field ofinterest. This field of interest might comprise also the complete imageand/or the complete first area. In this field of interest reference ismade to the pixels of a base plane of the captured image/images to buildup modification zones within the displayed output image. Instead of thebase plane reference to pixels of a vision sphere or of a cylinder canbe made onto which the captured images are projected to build up thedifferent parts of the displayed output image. In these modificationzones, especially the extension of the image is reduced in thewide-angle direction to adapt the first sector and/or the first imagearea to the second image area. One or more of such modification zonesmaybe provided. Such modification zones may also be provided in morethan one sector and/or area of the displayed output image such that 2 ormore areas are modified with respect to each other. Especially aplurality of image areas/sectors might comprise modification zones.

Different known projection methods can be used. A projection is amapping of one mathematical set, for example an image or a plane, ontoanother mathematical set. Among the well-known projections are forexample homography projections, rectangular projections, Mercatorprojections, gnomic projections, stereographic projections, equidistantprojections, orthographic projections and many more. The reference tothe base plane, the vision sphere and/or the cylinder is for examplecarried out such that the areas and/or sectors of the displayed outputimage, especially boundary parts of the areas and/or sectors aremathematically described by a homography projection onto the base plane,rectangular projection, Mercator projection or gnomic projection of thevision sphere or the base plane onto the output image plane.Alternatively, or additionally the reference to the base plane, thevision sphere and/or the cylinder is carried out by a, especially withregard to the image area and/or sector globally or locally weighted,vector sum, of one or more of the before described projections and/or astereographic projection, equidistant projection, orthographicprojection, a projection with a constant dihedral angle, a fish eyeprojection of the vision sphere or the base plane onto the output imageplane.

All operations described up to now as well as the operations describedin the following present a defined image whilst the vehicle is in motionand/or not moving. The image is adjusted depending on the application ofthe vehicle.

A method to produce a single image area from multiple, eventuallyoverlapping images, each provided from a different image capturingdevice, is now presented in detail.

Afterwards two methods to match the edges of two or more bordering imageareas, for example to simulate and display an image representing a planemirror with an additional mirror and creating a quite seamlesstransition between them, are also presented.

The methods are based on mapping a surface model of the environment tothe envisaged output image area via different well-known mappingfunctions.

As shown in FIG. 10, an image capturing device K_(i) (the subscript inumbers different image capturing devices) can be characterized by itstechnical projection center Z_(i), the captured image A_(i) and a knownmathematical projection model, the mapping function P_(i). The mappingfunction maps each line of sight S_(i), emerging from the technicalprojection center and directed to the environment, to exactly one points_(i) on the image area A_(i). The inverse mapping function P_(i) ⁻¹ isthen also known.

The environment seen by the image capturing devices can be modeled as a2-dimensional, arbitrarily formed, e.g. curved, surface F within the3-dimensional space. As can be seen in FIG. 11, each point x in the2-dimensional output image area B corresponds to a single point X on the3-dimensional surface F, related via the mapping function Q: X=Q(x).

When several imaging capturing devices are used, it is not a trivialtask to combine the images of individual image capturing devices to asingle image area and display it on a display without introducingerrors, ambiguities, distortions and/or general image mismatches whichcan distract or confuse the viewer.

However, it is the surprising finding of the present disclosure that,when several image capturing devices are positioned in such a way thattheir respective projection centers are very close to the same spacepoint, a common technical projection center {circumflex over (Z)} can beused to create a technically simplified and idealized projection model{circumflex over (P)}_(l) for each image capturing device K_(i) (thesubscript i denotes the different image capturing devices). Then eachline of sight Ŝ emerging from the common technical projection center{circumflex over (Z)}, directed to the environment and passing throughthe area of the image A_(i) is mapped on exactly one point

of the image A_(i). Each inverse mapping function P_(i) ⁻¹ can beanalogously transferred to an inverse mapping function {circumflex over(P)}_(i) ⁻¹, related to the common technical projection center{circumflex over (Z)}.

The method to produce a single image area from multiple, eventuallyoverlapping images, each provided from a different image capturingdevice, is performed using the following steps, as depicted in FIG. 12:

For each target point x in the 2-dimensional output image area B thevalue B(x), which can be for example a color or brightness value, has tobe generated. For this first the point X on the 3-dimensional surface Fis determined via the mapping function Q (X=Q(x)), as well as the lineof sight Ŝ emerging from the common technical projection center anddirected to the point X (Ŝ=S(X,{circumflex over (Z)})). Then for eachimage capturing device K_(i), which captured image A_(i), the mappingfunction {circumflex over (P)}_(l) is used to determine the position

(

={circumflex over (P)}_(l)(Ŝ)). This position can now be either withinor outside of the image A_(i). When inside a specific image A_(i), thecorresponding value A_(i)(

) is extracted. Depending on the technical realization, the value A_(i)(

) can for example be a scalar brightness value, a RGB vector or a vectorof a different color space. In case the position

does not have a corresponding value A_(i)(

) within the image A_(i), the value A_(i)(

) can be determined from existent neighboring position values, forexample by interpolation.

The value B(x) can now be determined by considering all the valuesA_(i)(

) which have been found. Different techniques and functions can beappropriate. For example, it can be appropriate to select a singleimaging device K_(j) and only take the corresponding value A_(j)(ŝ_(j)):B(x):=A_(j)(ŝ_(j)). But it can also be appropriate to use the weightedsum with a specific weight w_(i): B(x):=Σ_(i)w_(i)*A_(j)(

)

The first method to match the edges of two or more bordering image areasand to create a quite seamless transition between them, is exemplarilyrealized with two images in the following way and depicted in FIG. 13.

It is based on the adaption of neighboring parts of the bordering imageareas by defining an adapted mapping function.

Given two bordering image areas B₁ and B₂ with the common edge E, therespective surfaces F₁, F₂ and the mapping functions Q₁, Q₂. B is thearea to be matched, extends along the edge and consists of equal sizedparts B ₁⊂B₁, B ₂⊂B₂, such that B=B ₁∪B ₂, |B₁|=|B₂| and E⊂B. F ₁ (F ₂)denotes the part of F₁ (F₂) belonging to B ₁ (B ₂). Then one can definean adapted mapping function Q ₁ for the part B ₁:

${{\hat{Q}}_{1}(x)} = {{{w_{1}(x)}*{Q_{1}\left( {x_{0} + \frac{x - x_{0}}{2}} \right)}} + {{w_{2}(x)}*{Q_{2}\left( {x_{0} + \frac{x - x_{0}}{2} + h} \right)}}}$

Where the weight w₁(x) decreases gradually from 1.0 to 0.5 and w₂(x)gradually increases from 0.0 to 0.5 in the direction from B ₁ to B ₂ andfor all positions w₁(x)+w₂(x)=1 is valid. The 2-dimensional vector hrelates the “first” element of B ₁, x₀ with the first element of B ₂ inthe direction of adaption. In the same way, one can define an adaptedmapping function Q ₂ for the part B ₂:

${{\hat{Q}}_{2}(x)} = {{{w_{1}(x)}*{Q_{1}\left( {x_{0} + \frac{x - x_{0}}{2}} \right)}} + {{w_{2}(x)}*{Q_{2}\left( {x_{0} + \frac{x - x_{0}}{2} + h} \right)}}}$

with the same x₀ and h and wherein the weight w₁(x) decreases graduallyfrom 0.5 to 0.0 and w₂(x) gradually increases from 0.5 to 1.0 in thedirection from B ₁ to B ₂ and for all positions w₁(x)+w₂(x)=1 is valid.

In this way, the adapted mapping functions {circumflex over (Q)}₁ and{circumflex over (Q)}₂ implicitly define surfaces which depict agradually transiting 3-dimensional mixture of the surfaces F ₁ and F ₂.

This method can be straightforward used to adapt multiple image areas,when only independent edges exist, as shown in FIG. 14.

When multiple image areas with not only opposing edges should beadapted, on can easily first adapt all image areas in one direction andthen adapt the image areas along the second direction.

The second method to match the edges of two or more bordering imageareas and to create a quite seamless transition between them, isexemplarily realized with two images in the following way and depictedin FIGS. 15 and 16.

It is based on the adaption of a neighboring part of one of thebordering image areas and keeping the other image area fixed. For thisthe mapping function is adapted.

Given two bordering image areas B₁ and B₂ with the common edge E, therespective surfaces F₁, F₂ and the mapping functions Q₁, Q₂ and theinverse mapping function {circumflex over (Q)}₂ ⁻¹. Without the loss ofgenerality B₁ is chosen to be fixed and B₂ will be adapted. For eachx=(u,v)∈E there exists also a x′=(u,v′)∈E with the same horizontalcoordinate u but in general different vertical coordinate v′≠v, suchthat S(Q₁(x),{circumflex over (Z)})=S(Q₂(x′),{circumflex over (Z)}).

By now moving identifying the column v′ from the original description ofimage area B₂ and moving it to the adapted area of image area B₂, asnearly seaming less transition between the two image areas can bereached. For this first the position X on the surface F₁ is identifiedby X=Q₁(x). This position is on the line of sight Ŝ=S(X,{circumflex over(Z)}). Because the surface F₂ is also defined at this point, thecrossing point X′ of Ŝ with F₂. With the help of the inverse mappingfunction the point x′ in B₂ can be calculated: x′=Q₂ ⁻¹(X′). Therefore,a function ƒ can be defined, assigning each point {tilde over(x)}=(ũ,{tilde over (v)}) from B₂ den the point {tilde over(x)}′=(ũ,{tilde over (v)}′) in with the same ũ, but from the column{tilde over (v)}′. In this way, the content of the column {tilde over(v)} of B₁ is “extended”.

The adapted mapping function {circumflex over (Q)}₂ can now be definedas {circumflex over (Q)}₂ (x)=Q₂(ƒ(x)).

When faced with the task to match several image areas, it is clear thatone image area cannot be adapted to two opposite bordering image areasat the same time. Therefore, it is necessary to consecutively match theimage areas. This also works for matching image areas in a 2-dimensionalmatrix arrangement, but it has to be taken care not to create asituation, in which an image area which is to be adapted, already hastwo identified neighboring image areas on opposing edges. In general,the sequence, in which the areas will be adapted, will lead to differentend results.

The before described modification might be applied alternatively or inaddition to a second sector of the second area being located adjacent tothe first sector of the first area.

The implementations disclosed herein also relate to an object detectionand classification system for use in a variety of contexts. The presentdisclosure contains feature descriptor that combines informationrelating to what a feature is with information relating to where thefeature is located, especially within the captured image, with respectto a query point. This feature descriptor provides advantages over priorfeature descriptors because, by combining the “what” with the “where,”it reduces the resources needed to detect and classify an object becausea single descriptor can be used instead of multiple feature descriptors.The resulting system therefore is more efficient than prior systems, andcan more accurately detect and classify objects in situations wherehardware and/or software resources are limited.

FIG. 17 illustrates a rear view of a vehicle 112 with an objectdetection and classification system 110 according to the presentdisclosure. The vehicle 112 includes an image sensor 114 to capture animage 116 of the environment surrounding the vehicle 112. The image mayinclude a range of view through an angle 118, thus the image 116 maydepict only a portion of the area surrounding the vehicle 112 as definedby the angle 118. The image 116 may include an object 120. The object120 may be any physical object in the environment surrounding thevehicle 112, such as a pedestrian, another vehicle, a bicycle, abuilding, road signage, road debris, etc. The object detection andclassification system 110 may assign a classification to the object 120.The classification may include the type of road object, whether theobject is animate or inanimate, whether the object is likely to suddenlychange direction, etc. The object detection and classification system110 may further assign a range of characteristics to the object 120 suchas a size, distance, a point representing the center of the object, avelocity of the object, an expected acceleration range, etc.

The image sensor 114 may be various types of optical image sensors,including without limitation a digital camera, a range finding camera, acharge-coupled device (CCD), a complementary metal oxide semiconductor(CMOS) sensor, or any other type of image sensor capable of capturingcontinuous real-time images. In an implementation, the vehicle 112 hasmultiple image sensors 114, each image sensor 114 may be positioned soas to provide a view of only a portion of the environment surroundingthe vehicle 112. As a group, the multiple image sensors 114 may covervarious views from the vehicle 112, including a front view of objects inthe path of the vehicle 112, a rear-facing image sensor 114 forcapturing images 116 of the environment surrounding the vehicle 112including objects behind the vehicle 112, and/or side-facing imagesensors 114 for capturing images 116 of object next to or approachingthe vehicle 112 from the side. In an implementation, image sensors 112may be located on various parts of the vehicle. For example, withoutlimitation, image sensors 112 may be integrated into an exterior mirrorof the vehicle 112, such as on the driver's exterior side mirror 122.Alternatively, or additionally, the image sensor 112 may be located onthe back of the vehicle 112, such as in a rear-light unit 124. The imagesensor 112 may be forward-facing and located in the interior rear-viewmirror, dashboard, or in the front headlight unit of the vehicle 112.

Upon capture of an image 116 of the environment surrounding the vehicle112, the object detection and classification system 110 may store theimage 116 in a memory and perform analysis on the image 116. One type ofanalysis performed by the object detection and classification system 110on the image 116 is the identification of keypoints and associatedkeypoint data. Keypoints, also known as interest points, are spatiallocations or points in the image 116 that define locations that arelikely of interest. Keypoint detections methods may be supplied by athird-party library, such as the SURF and FAST methods available in theOpenCV (Open Source Computer Vision) library. Other methods of keypointdetection include without limitation SIFT (Scale-Invariant FeatureTransform). Keypoint data may include a vector to the center of thekeypoint describing the size and orientation of the keypoint, and visualappearance, shape, and/or texture in a neighborhood of the keypoint,and/or other data relating to the keypoint.

A function may be applied to a keypoint to generate a keypoint value. Afunction may take a keypoint as a parameter and calculate somecharacteristic of the keypoint. As one example, a function may measurethe image intensity of a particular keypoint. Such a function may berepresented as f(zk), where f is the image intensity function and zk isthe kth keypoint in an image. Other functions may also be applied, sucha visual word in a visual word index.

FIG. 18 illustrates a schematic diagram 200 of an image capture 204taken by an image sensor 202 on a vehicle. The image capture 204includes a query point (xc, yc) and a plurality of keypoints z0-z4. Aquery point is a point of interest that may or may not be a keypoint,for which the object detection and classification system may choose forfurther analysis. In an implementation, the object detection andclassification system may attempt to determine whether a query point isthe center of an object to assist in classification of the object.

Points in the image capture 204 may be described with reference to aCartesian coordinate system wherein each point is represented by anordered pair, the first digit of the pair referring to the point'sposition along the horizontal or x-axis, and the second digit of thepair referring to the point's position along the vertical or y-axis. Theorientation of the horizontal and vertical axes with respect to theimage 204 is shown by the axis 206. Alternatively, points in the imagecapture 204 may be referred to with complex numbers where each point isdescribed in the form x+iy where i=√(−1). In another implementation, aquery point may serve as the origin of a coordinate system, and thelocations of keypoints relative to the query point may be described asvectors from the query point to each of the keypoints.

The image detection and classification system 110 uses a new descriptorfunction, to produce an evaluation of a query point in an image 204 thatcombines a representation of what the feature is and where the featureis located in relation to the query point into a single representation.For any image 204 with a set of keypoints z0-z4 in the neighborhood of aquery point (xc, yc), the descriptor for the query point is as follows:

$\begin{matrix}{{F(\zeta)} = {\frac{1}{N}{\sum\limits_{k = 0}^{N - 1}{\left( {z_{k} - z_{c}} \right)e^{i\; 2\pi\;{f{(z_{k})}}\zeta}}}}} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

where N is the number of keypoints in the image from the environmentsurrounding the vehicle in the neighborhood of the query point, zc isthe query point represented in complex space, zk is the kth keypoint,f(zk) is the feature value of the kth keypoint, and ζ is the continuousindependent variable of the descriptor function F(ζ).

To obtain a descriptor that is invariant to scale and orientation,Equation (1) may be modified by letting zm be the mean value of zkvalues:

$\begin{matrix}{z_{m} = {\frac{1}{N}{\sum\limits_{k = 0}^{N - 1}\left( {z_{k} - z_{c}} \right)}}} & \left( {{Equation}\mspace{14mu} 2} \right)\end{matrix}$

By dividing the right-hand side of Equation (1) by |zm|, a scaleinvariant version of the descriptor is obtained. On the other hand, bydividing both sides of Equation (1) by

$\frac{z_{m}}{z_{m}}$a rotation-invariant version of the descriptor is obtained. In order towrite a descriptor that is invariant in both scale and orientation,dividing by zm yields the following descriptor:

$\begin{matrix}{{F(\zeta)} = {\frac{1}{z_{m}}{\sum\limits_{k = 0}^{N - 1}{\left( {z_{k} - z_{c}} \right)e^{i\; 2\pi\;{f{(z_{k})}}\zeta}}}}} & \left( {{Equation}\mspace{14mu} 3} \right)\end{matrix}$

The division by N is omitted from Equation (3) since the contribution ofthe keypoint number is already neutralized through the division by zm.Due to the similarity of Equation (3) to the formula for the InverseFourier Series, Equation (3) may be referred to herein as a Fourier Fan.

Since Equation (3) is a function of a continuous variable ζ, it may besampled for use in the object detection and classification system 100.In an implementation, a sampling frequency greater than 2max(f) may bechosen where max( ) indicates the maximum value of the function f.Another characteristic of Equation (3) is that it is infinite over thedomain of the variable ζ. Sampling an infinite equation will result inan infinite number of samples, which may not be practical for use in theobject detection and classification system 100. If Equation (3) is aperiodic function, however, then it would be sufficient to sample oneonly a single period of Equation (3), and to ignore the remainingperiods. In an implementation, Equation (3) is made to be periodic byrequiring all values of the function f to be integer multiples of asingle frequency f0. As such, for Equation (3) to be able to be sampled,the function f must have a known maximum, and for the Equation (3) to beperiodic, the function f must be quantized such that the values of f areinteger multiples of f0.

In an implementation, the function f may represent more than a simplefeature, such as the image intensity. Instead, the function f may be adescriptor function of each of the keypoints, such as those referred toherein (e.g., SIFT and/or SURF descriptors). Such descriptors areusually not simple scalar values, but rather are more likely to be highdimensional feature vectors, which cannot be incorporated directly inEquation (3) in a trivial manner. It is, however, possible toincorporate complex descriptors as feature values by clustering thedescriptors in an entire set of training data and to use the index ofthe corresponding cluster as the value for f. Such cluster centers maybe referred to as “visual words” for f. Let fk be the descriptor for akeypoint k, if fk takes integer values, e.g., 3, then there is adescriptor at the keypoint located at zk-zc, which can be assigned tocluster 3. It should be appreciated that, in this example, f isquantized and the number of clusters is the function's maximum which isknown. These characteristics are relevant because they are thecharacteristics of f needed to make Equation (3) able to be sampled andperiodic.

In an implementation, an order is imposed on the visual word clustercenters, such that the output of f is not a categorical value. In otherwords, without an order, the distance between cluster 2 and cluster 3 isnot necessarily less than the distance between cluster 2 and cluster 10because the numerical values are merely identifiers for the clusters. Anorder for the visual words may be imposed using multidimensional scaling(MDS) techniques. Using MDS, one can find a projection into a lowdimensional feature space from a high dimensional feature space suchthat distances in the low dimensional feature space resemble as much aspossible distances in the high dimensional feature space. Applied to thevisual words using MDS, the cluster centers may be projected into aone-dimensional space for use as a parameter for f. In oneimplementation, a one-dimensional feature space is chosen as the lowdimensional feature space because one dimensional space is the onlyspace in which full ordering is possible.

The object detection and classification system may be tuned according toa set of training data during which parameters for the system may bechosen and refined. For example, descriptor values and types may bechosen, the size of the neighborhood around a query point may be set,the method of choosing keypoints, the number of keypoints chosen perimage, etc. may also be chosen. Since the tuning of the object detectionand classification system is a type of machine learning, it may besusceptible to a problem known as “overfitting.” Overfitting manifestsitself when machine classifiers over-learn the training data leading tomodels which do not generalize well on other data, the other data beingreferred to herein as “test data.” In the descriptor of Equation (3),overfitting could occur if, on training data, the object detection andclassification system overfits the positions of the keypoints withrespect to the query point. Changes in the positions of the keypointsthat are not present in training data, which could occur due to noiseand intra-class variance, will not always be handled well by the objectdetection and classification system when acting on test data. To addressthe issue of overfitting, at each query point (xc, yc), instead ofextracting a single Fourier Fan Equation (3) on training data, multiplerandom Fans may be extracted, denoted by the set Mf (e.g., 15f). Each ofthe random Fans contains only a subset of the available N keypoints inthe neighborhood of the query point (xc, yc). Later, when the objectdetection and classification system is running on test data, the sameset Mf of random Fourier Fans is extracted, and the result is confirmedaccording to majority agreement among the set of random Fourier Fans.Random Fourier Fans also allow the object detection and classificationsystem to learn from a small number of images since several featurevectors are extracted at each object center.

In the comparison of Equation (3), the “Fourier Fan,” to the formula forthe inverse Fourier Series, it should be understood that there are somedifferences between the two. For example, only those frequencies thatbelong to the neighborhood of a query point are available for eachFourier Fan. As another example, shifting all coefficients zk by aconstant za, i.e. a shift of the object center, is not equivalent toadding a Dirac impulse in the ζ domain, even if it is assumed that thesame keypoints are available in the new query point neighborhood. Thisis true because the addition of za is not a constant everywhere, butonly to the available frequencies, and zero for the other frequencies.

FIG. 19 illustrates a block diagram of an object detection andclassification system 300 that may be useful for the implementationsdisclosed herein. The object detection and classification system 300includes an image sensor 302 directed at the environment surrounding avehicle. The image sensor 302 may capture images of the environmentsurrounding the vehicle for further analysis by the object detection andclassification system 300. Upon capture, an image from the environmentsurrounding a vehicle may be stored in the memory 304. The memory 304may include volatile or non-volatile memory and may store imagescaptured by the image sensor as well as data produced by analysis of theimages captured by the image sensor. A processor 306 may carry outoperations on the images stored in memory 304. The memory 304 may alsostore executable program code in the form of program modules that may beexecuted by the processor 306. Program modules stored on the memory 304include without limitation, hazard detection program modules, imageanalysis program modules, lens obstruction program modules, blind spotdetection program modules, shadow detection program modules, trafficsign detection program modules, park assistance program modules,collision control and warning program modules, etc.

The memory 304 may further store parameters and settings for theoperation of the object detection and classification system 300. Forexample, parameters relating to the training data may be stored on thememory 304 including a library of functions f and keypoint settings forcomputation and calculation of Random Fourier Fans. The memory 304 mayfurther be communicatively coupled to extracting circuitry 308 forextracting keypoints from the images stored on the memory 304. Thememory 304 may further be communicatively coupled to query pointevaluation circuitry 310 for taking image captures with keypoints andassociated keypoint data and evaluating the images with keypoints andkeypoint data according to Fourier Fans to produce sampled Fourier Fanvalues.

If the sampled Fourier Fan values produced by the query point evaluationcircuitry 310 meet a potential hazard condition, then signal buscircuitry 312 may send a signal to an alert system 314 and/or a vehiclecontrol system 316. Sampled Fourier Fan values may first be processed byone or more program modules residing on memory 304 to determine whetherthe sampled values meet a potential hazard condition. Examples ofsampled values that may meet a potential hazard condition are an objectdetermined to be a collision risk to the vehicle, an object that isdetermined to be a vulnerable road user that is at risk of being struckby the vehicle, a road sign object that indicates the vehicle istraveling in the wrong part of a road or on the wrong road, objects thatindicate a stationary object that the vehicle might strike, objects thatrepresent a vehicle located in a blind spot of the operator of thevehicle.

If the sampled values of a Fourier Fan function satisfy a potentialhazard condition, the signal bus circuitry 312 may send one or moresignals to the alert system 314. In an implementation, signals sent tothe alert system 312 include acoustic warnings to the operator of thevehicle. Examples of acoustic warnings include bells or beep sounds,computerized or recorded human language voice instructions to theoperator of the vehicle to suggest a remedial course of action to avoidthe cause the of sample value meeting the potential hazard condition. Inanother implementation, signals sent to the alert system 314 includetactile or haptic feedback to the operator of the vehicle. Examples oftactile or haptic feedback to the operator of the vehicle includewithout limitation shaking or vibrating the steering wheel or controlstructure of the vehicle, tactile feedback to the pedals, such as apedal that, if pushed, may avoid the condition that causes the samplevalue of the Fourier Fan to meet the potential hazard condition,vibrations or haptic feedback to the seat of the driver, etc. In anotherimplementation, signals sent to the alert system 314 include visualalerts displayed to the operator of the vehicle. Examples of visualalerts displayed to the operator of the vehicle include lights orindications appearing on the dashboard, heads-up display, and/or mirrorsvisible to the operator of the vehicle. In one implementation, thevisual alerts to the operator of the vehicle include indications ofremedial action that, if taken by the operator of the vehicle, may avoidthe cause of the sample value of the Fourier Fan meeting the potentialhazard condition. Examples of remedial action, include an indication ofanother vehicle in the vehicle's blind spot, an indication that anothervehicle is about to overtake the vehicle, an indication that the vehiclewill strike an object in reverse that may not be visible to the operatorof the vehicle, etc.

In another implementation, if the sampled values of a Fourier Fanfunction satisfy a potential hazard condition, the signal bus circuitry312 may send one or more signals to the vehicle control system 316.Examples of signals sent to the vehicle control system 316 includesignals to the steering system to alter the direction of the vehicle inan attempt to avoid the object that is the cause of the sampled valuesof the Fourier Fan function to satisfy the potential hazard condition.In another implementation, a signal sent to the vehicle control system316 may include signals to sound the horn of the vehicle to alert theobject that caused the sample values of the Fourier Fan function tosatisfy the hazard condition that the vehicle with the object detectionand classification system is present. In yet another implementation, thesignal sent to the vehicle control system 316 include a signal to engagethe brakes of the vehicle to avoid a collision with the detected object.

FIG. 20 illustrates example operations 400 for detecting and classifyingan object and outputting a signal if a sample value of a Fourier Fanmeets a potential hazard condition. A capturing operation 402 capturesan image from the environment surrounding a system with an image sensor,the image from the environment surrounding the system having one or morekeypoints. The image from the environment surrounding the system may becaptured by an image sensor mounted anywhere on the system. In oneimplementation, the system is a vehicle. The captured image may bestored in a memory in an object detection and classification system.Once stored, a processor may execute a keypoint detection program moduleto identify the keypoints. Alternatively, or additionally, extractingcircuitry may identify the keypoints in the image. The keypoints may beidentified according to a number of methods, such as methods provided bythird party libraries, and data parameters for the methods, such as thenumber of keypoints to extract or conditions for a keypoint extractionmay be stored on the memory in the object detection and classificationsystem.

Extracting operation 404 extracts at least one feature value from atleast one of the one or more keypoints in the image from the environmentsurrounding the system. A processor may execute a feature valueextraction program module to extract the features values of keypoints.Alternatively, or additionally, extracting circuitry may extract thefeature values of keypoints in the stored image. The feature value of akeypoint may be determined according to a function f. In someembodiments, the function f may have certain characteristics for use ina Fourier Fan Equation: having a known maximum, such that a samplingrate may be set to 2max(f), and being periodic, such that only oneperiod of f need be sampled.

Sampling operation 406 samples a periodic descriptor function for aquery point in the image from the environment surrounding the system toproduce a sample value, the periodic descriptor function depending onthe distance between at least one of the one or more keypoints and thequery point in complex space, the periodic descriptor function furtherdepending on at least one feature value extracted from at least one ofthe one or more keypoints in the image from the environment surroundingthe vehicle. In an implementation, the sampled periodic descriptorfunction in sampling operation 406 is Equation (3).

At decision block 408, if the sample value of the periodic descriptorfunction does not meet a potential hazard condition, then the methodends or returns to capturing operation 402. If the sample value of theperiodic descriptor function does meet a potential hazard condition,then outputting operation 410 sends a signal via a signal bus. Thesignal sent via the signal bus in outputting operation 410 may be to analert system of a vehicle, which may display an acoustic or visualsignal to the operator of the vehicle regarding the object that is thecause of the sample value exceeding the potential hazard condition. Inanother embodiment, the signal bus sends at outputting operation 410 asignal to a vehicle control system. The system sent to the vehiclecontrol system may cause the vehicle to change speed or directionwithout intervention from the operator of the vehicle to avoid acollision with the object that caused the sample value to exceed thepotential hazard condition.

FIG. 21 discloses a block diagram of a processor system 500 suitable forimplementing one or more aspects of an object detection andclassification system with Fourier fans. The processor system 500 iscapable of executing a processor program product embodied in a tangibleprocessor-readable storage medium. Data and program files may be inputto the processor system 500, which reads the files and executes theprograms therein using one or more processors. Some of the elements of aprocessor system 500 are shown in FIG. 21 wherein a processor 502 isshown having an input/output (I/O) section 504, a Central ProcessingUnit (CPU) 506, and a memory section 508. There may be one or moreprocessors 502, such that the processor 502 of the computing system 500comprises a single central-processing unit 506, or a plurality ofprocessing units. The processors may be single core or multi-coreprocessors. The described technology is optionally implemented insoftware loaded in memory 508, a disc storage unit 512, and/orcommunicated via a wired or wireless network link 514 on a carriersignal (e.g., Ethernet, 3G wireless, 5G wireless, LTE (Long TermEvolution)) thereby transforming the processing system 500 in FIG. 21 toa special purpose machine for implementing the described operations. Thedisc storage unit 512 may include volatile memory, non-volatile memory,solid state memory, flash memory, hybrid, and/or traditional magneticrotating data storage media.

The I/O section 504 may be connected to one or more user-interfacedevices (e.g., a mobile device, a touch-screen display unit 518, etc.)or the disc storage unit 512. Processor program products containingmechanisms to effectuate the systems and methods in accordance with thedescribed technology may reside in the memory section 508 or on thestorage unit 512 of such a system 500.

A communication interface 524 is capable of connecting the processorsystem 500 to an enterprise network via the network link 514, throughwhich the processor system 500 can receive and/or send instructions anddata embodied in a carrier wave. The communication interface 524 mayreceive signals and data representing an image from the environmentsurrounding the system via an image sensor. The communication interfacemay send signals from the processor system 500 to the image detectionand classification system including without limitation audible alertsignals sent to the operator of a vehicle, data signals sent to a HUD(heads-up-display) visible to the operator of a vehicle, a video screensuch as display 518 that is visible to the operator of a vehicle fordisplay of information regarding objects and/or alerts, a vehiclecontrol signal for altering the control of a vehicle to avoid a hazard,etc.

When used in a local area networking (LAN) environment, the processorsystem 500 is connected (by wired connection or wirelessly) to a localnetwork through the communication interface 524, which is one type ofcommunications device. When used in a wide-area-networking (WAN)environment, the processor system 500 typically includes a modem, anetwork adapter, or any other type of communications device forestablishing communications over the wide area network. In a networkedenvironment, program modules depicted relative to the processor system500 or portions thereof, may be stored in a remote memory storagedevice. It is appreciated that the network connections shown areexamples of communications devices for, and other means of, establishinga communications link between the processor and other devices may beused.

In an example implementation, a network agent of an enterprise networkmay be embodied by instructions stored in memory 508 and/or the storageunit 512 and executed by the processor 502. Further, local computingsystems, remote data sources and/or services, and other associated logicrepresent firmware, hardware, and/or software, which may be configuredto transmit data outside of the enterprise network via the networkagent. The network agent of the enterprise network may be implementedusing a general-purpose computer and specialized software (such as aserver executing service software), a special purpose computing systemand specialized software (such as a mobile device or network applianceexecuting service software), or other computing configurations. Inaddition, data-selection policies and data transmission preparationpolicies may be stored in the memory 508 and/or the storage unit 512 andexecuted by the processor 502.

An example system includes an image sensor mounted on a system andconfigured to capture an image from the environment surrounding thesystem, the image from the environment surrounding the system includingone or more keypoints, extracting circuitry configured to extract atleast one feature value from at least one of the one or more keypointsin the image from the environment surrounding the system, query pointevaluation circuitry communicatively coupled to the image sensor and theextracting circuitry and configured to classify an object by sampling aperiodic descriptor function for a query point in the image from theenvironment surrounding the system to produce a sample value, theperiodic descriptor function depending on the distance between at leastone of the one or more keypoints and the query point in complex space,the periodic descriptor function further depending on at least onefeature value extracted from at least one of the one or more keypointsin the image from the environment surrounding the system, and a signalbus configured to transmit a signal to the operator of the system if thesample value satisfies an alert condition.

An example system of any preceding system includes that the periodicdescriptor function is defined as follows:

${F(\zeta)} = {\frac{1}{Z}{\sum\limits_{k = 0}^{N - 1}{\left( {z_{k} - z_{c}} \right)e^{i\; 2\pi\;{f{(z_{k})}}\zeta}}}}$where N is the number of keypoints in the image from the environmentsurrounding the system in the neighborhood of the query point, zc is thequery point represented in complex space, zk is the kth keypoint, f(zk)is the feature value of the kth keypoint, and ζ is the continuousindependent variable of the descriptor function F(ζ).

An example system of any preceding system includes that the periodicdescriptor function is a scale-invariant or rotation-invariant functiondefined as follows:

${F(\zeta)} = {\frac{1}{z_{m}}{\sum\limits_{k = 0}^{N - 1}{\left( {z_{k} - z_{c}} \right)e^{i\; 2\pi\;{f{(z_{k})}}\zeta}}}}$wherein:

$z_{m} = {\frac{1}{N}{\sum\limits_{k = 0}^{N - 1}{\left( {z_{k} - z_{c}} \right).}}}$

An example system of any preceding system includes the signaltransmitted to the operator of the system is an audible alert.

An example system of any preceding system includes the signaltransmitted to the operator of the system is a visual alert.

An example system of any preceding system includes the signaltransmitted to the operator of the system includes haptic feedback.

An example system of any preceding system includes the at least onefeature value includes a visual appearance feature.

An example system of any preceding system includes the at least onefeature value includes a visual word index in a dictionary of visualwords, the visual word index having an order relation determined bymulti-dimensional scaling.

An example system of any preceding system includes the feature values ofthe kth keypoint are integer multiples of a single frequency f0.

An example system of any preceding system includes the query pointevaluation circuitry is further configured to sample a plurality ofrandomized descriptor functions, each randomized descriptor functionincluding only a subset of the one or more keypoints.

An example system of any preceding system includes that the system is avehicle.

An example system of any preceding system includes that the signal tothe system is a vehicle control signal.

An example system includes means for capturing an image from theenvironment surrounding a system with an image sensor, the image fromthe environment surrounding the system having one or more keypoints. Theexample system further includes means for extracting at least onefeature value from at least one of the one or more keypoints in theimage from the environment surrounding the system. The example systemfurther includes means for sampling a periodic descriptor function for aquery point in the image from the environment surrounding the system toproduce a sample value to classify an object, the periodic descriptorfunction depending on the distance between at least one of the one ormore keypoints and the query point in complex space, the periodicdescriptor function further depending on at least one feature valueextracted from at least one of the one or more keypoints in the imagefrom the environment surrounding the system. The example system furtherincludes means for outputting an alert signal via a signal bus if thesample value satisfies an alert condition.

An example method includes capturing an image from the environmentsurrounding a system with an image sensor, the image from theenvironment surrounding the system having one or more keypoints,extracting at least one feature value from at least one of the one ormore keypoints in the image from the environment surrounding the system,sampling a periodic descriptor function for a query point in the imagefrom the environment surrounding the system to produce a sample value toclassify an object, the periodic descriptor function depending on thedistance between at least one of the one or more keypoints and the querypoint in complex space, the periodic descriptor function furtherdepending on at least one feature value extracted from at least one ofthe one or more keypoints in the image from the environment surroundingthe system, and outputting an alert signal via a signal bus if thesample value satisfies an alert condition.

An example method of any preceding method includes that the periodicdescriptor function is defined as follows:

${F(\zeta)} = {\frac{1}{N}{\sum\limits_{k = 0}^{N - 1}{\left( {z_{k} - z_{c}} \right)e^{i\; 2\pi\;{f{(z_{k})}}\zeta}}}}$where N is the number of keypoints in the image from the environmentsurrounding the system in the neighborhood of the query point, zc is thequery point represented in complex space, zk is the kth keypoint, f(zk)is the feature value of the kth keypoint, and ζ is the continuousindependent variable of the descriptor function F(ζ).

An example method of any preceding method includes that the alert signalresults in an audible alert.

An example method of any preceding method includes that the alert signalresults in a visual alert.

An example method of any preceding method includes that the alert signalresults in haptic feedback to the operator of the system.

An example method of any preceding method includes that the at least onefeature includes an image intensity.

An example method of any preceding method includes that the at least onefeature value includes a visual word index in a dictionary of visualwords.

An example method of any preceding method includes that the visual wordindex has an order relation determined by multi-dimensional scaling.

An example method of any preceding method includes that the featurevalues of the kth keypoint are integer multiples of a single frequencyf0.

An example method of any preceding method includes that the periodicdescriptor function is a scale-invariant or rotation-invariant functiondefined as follows:

${F(\zeta)} = {\frac{1}{z_{m}}{\sum\limits_{k = 0}^{N - 1}{\left( {z_{k} - z_{c}} \right)e^{i\; 2\pi\;{f{(z_{k})}}\zeta}}}}$wherein:

$z_{m} = {\frac{1}{N}{\sum\limits_{k = 0}^{N - 1}{\left( {z_{k} - z_{c}} \right).}}}$

An example method of any preceding method includes that the samplingoperation includes sampling a plurality of randomized descriptorfunctions, each randomized descriptor function including only a subsetof the one or more keypoints.

An example system includes an image sensor mounted on a system andconfigured to capture an image from the environment surrounding thesystem, the image from the environment surrounding the system includingone or more keypoints, one or more processors, and a memory storingprocessor-executable instructions to perform the operations of:extracting, by the processor, at least one feature value from at leastone of the one or more keypoints in the image from the environmentsurrounding the system; sampling, by the processor, a periodicdescriptor function for a query point in the image from the environmentsurrounding the system to produce a sample value to classify an object,the periodic descriptor function depending on the distance between atleast one of the one or more keypoints and the query point in complexspace, the periodic descriptor function further depending on at leastone feature value extracted from at least one of the one or morekeypoints in the image from the environment surrounding the system, andoutputting, by the processor, an alert signal via a signal bus if thesample value satisfies an alert condition.

An example system of any preceding system includes that the periodicdescriptor function is defined as follows:

${F(\zeta)} = {\frac{1}{N}{\sum\limits_{k = 0}^{N - 1}{\left( {z_{k} - z_{c}} \right)e^{i\; 2\pi\;{f{(z_{k})}}\zeta}}}}$where N is the number of keypoints in the image from the environmentsurrounding the system in the neighborhood of the query point, zc is thequery point represented in complex space, zk is the kth keypoint, f(zk)is the feature value of the kth keypoint, and ζ is the continuousindependent variable of the descriptor function F(ζ).

An example system of any preceding system includes that the alert signalresults in an audible alert.

An example system of any preceding system includes that the alert signalresults in a visual alert.

An example system of any preceding system includes that the alert signalresults in haptic feedback to the operator of the system.

An example system of any preceding system includes that the at least onefeature includes an image intensity.

An example method of any preceding method includes that the at least onefeature value includes a visual word index in a dictionary of visualwords.

An example method of any preceding method includes that the visual wordindex has an order relation determined by multi-dimensional scaling.

An example system of any preceding system includes that the featurevalues of the kth keypoint are integer multiples of a single frequencyf0.

An example system of any preceding system includes that the periodicdescriptor function is a scale-invariant or rotation-invariant functiondefined as follows:

${F(\zeta)} = {\frac{1}{z_{m}}{\sum\limits_{k = 0}^{N - 1}{\left( {z_{k} - z_{c}} \right)e^{i\; 2\pi\;{f{(z_{k})}}\zeta}}}}$wherein:

$z_{m} = {\frac{1}{N}{\sum\limits_{k = 0}^{N - 1}{\left( {z_{k} - z_{c}} \right).}}}$

An example system of any preceding system includes the samplingoperation includes sampling a plurality of randomized descriptorfunctions, each randomized descriptor function including only a subsetof the one or more keypoints.

The implementations described herein may be implemented asprocessor-implemented methods, with circuitry such as ASIC designs, orany combination thereof. The system described herein may include avariety of tangible computer-readable storage media and intangiblecomputer-readable communication signals. Tangible computer-readablestorage can be embodied by any available media that can be accessed bythe object detection and classification system and includes bothvolatile and nonvolatile storage media, removable and non-removablestorage media. Tangible computer-readable storage media excludesintangible communications signals and includes volatile and nonvolatile,removable and non-removable storage media implemented in any method ortechnology for storage of information such as computer readableinstructions executable by a processor, data structures, program modulesor other data. Tangible computer-readable storage media includes, but isnot limited to, RAM, ROM, EEPROM, flash memory or other memorytechnology, CDROM, digital versatile disks (DVD) or other optical diskstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or any other tangible medium which canbe used to store the desired information and which can be accessed bythe object detection and classification system. In contrast to tangiblecomputer-readable storage media, intangible computer-readablecommunication signals may embody computer readable instructionsexecutable by a processor, data structures, program modules or otherdata resident in a modulated data signal, such as a carrier wave orother signal transport mechanism. The term “modulated data signal” meansa signal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, intangible communication signals include wired mediasuch as a wired network or direct-wired connection, and wireless mediasuch as acoustic, RF, infrared and other wireless media.

Some embodiments may comprise an article of manufacture. An article ofmanufacture may comprise a tangible storage medium to store logic.Examples of a storage medium may include one or more types ofcomputer-readable storage media capable of storing electronic data,including volatile memory or non-volatile memory, removable ornon-removable memory, erasable or non-erasable memory, writeable orre-writeable memory, and so forth. Examples of the logic may includevarious software elements, such as software components, programs,applications, computer programs, application programs, system programs,machine programs, operating system software, middleware, firmware,software modules, routines, subroutines, functions, methods, procedures,software interfaces, application program interfaces (API), instructionsets, computing code, computer code, code segments, computer codesegments, words, values, symbols, or any combination thereof. In oneembodiment, for example, an article of manufacture may store executablecomputer program instructions that, when executed by a computer, causethe computer to perform methods and/or operations in accordance with thedescribed embodiments. The executable computer program instructions mayinclude any suitable type of code, such as source code, compiled code,interpreted code, executable code, static code, dynamic code, and thelike. The executable computer program instructions may be implementedaccording to a predefined computer language, manner or syntax, forinstructing a computer to perform a certain function. The instructionsmay be implemented using any suitable high-level, low-level,object-oriented, visual, compiled and/or interpreted programminglanguage.

The implementations described herein are implemented as logical steps inone or more computer systems. The logical operations may be implemented(1) as a sequence of processor-implemented steps executing in one ormore computer systems and (2) as interconnected machine or circuitmodules within one or more computer systems. The implementation is amatter of choice, dependent on the performance requirements of thecomputer system being utilized. Accordingly, the logical operationsmaking up the implementations described herein are referred to variouslyas operations, steps, objects, or modules. Furthermore, it should beunderstood that logical operations may be performed in any order, unlessexplicitly claimed otherwise or a specific order is inherentlynecessitated by the claim language. The above specification, examples,and data, together with the attached appendices, provide a completedescription of the structure and use of exemplary implementations

A vehicle comprising displaying devices, processing devices and sensorssuch as cameras is also described in this disclosure. In or on thevehicle different displaying devices, processing devices and cameras asdescribed before as well as additional devices and cameras can beinstalled, configured and interconnected.

The displaying devices can be mounted inside or outside the vehicle andcan be used to transmit optical information to the driver and/or anyperson or object inside and outside of the vehicle. The display devicescan also be configured to transmit information via haptics, acoustics,odors, chemicals and/or other forms of electromagnetic fields. Theinformation is typically first collected from sensors and other signalreceiving devices on or in the vehicle and then processed by processingdevices. A multitude or only one processing device can be installed inthe vehicle to process the pictures and information provided by thecameras and sensors. Optionally the processing devices can be remotelylocated and the vehicle is wirelessly connected to the remote processingunit. The processed information is then directed to the differentdisplaying devices to inform the driver and/or any person or objectinside and outside of the vehicle. Depending on the location of thedisplaying devices and the nature of the receiver, the output ofdifferent information with different output means is induced.

The display devices can also be configured to receive input from thedriver and/or any person or object inside and outside of the vehicle.This input can be received via different sensing means, comprising forexample photosensitive sensors, acoustic sensors, distance sensors,touch-sensitive surfaces, temperature sensors, pressure sensors, odordetectors, gas detectors and/or sensors for other kind ofelectromagnetic fields. This input can be used to control or change thestatus of the output of the displaying device and/or other components onor in the vehicle. For example, the field of view, the contrast, thebrightness and/or the colors displayed on the display device, but alsothe strength of the touch feedback, sound volume and other adjustableparameters can be changed. As further examples, the position or focus ofa camera, the temperature or lighting inside the vehicle, the status ofa mobile device, like a mobile phone, carried by a passenger, the statusof a driver assistance system or the stiffness of the suspension can bechanged. Generally, every adjustable parameter of the vehicle can bechanged.

Preferably the information from the sensing means is first processed bya (sensing) processing device, but it can also be directly processed bythe sensor means or the display device comprising a processing device.Preferably the display device comprises a multi-touch display so thatthe driver or any other passenger can directly react to opticalinformation delivered by the display device by touching specific areason the display. Optionally gestures, facial expression, eye movement,voice, sound, evaporations, breathing and/or postural changes of thebody can also be detected, for example via an optical sensory means likea camera, and used to provide contact-free input to also control thedisplay device.

Information stemming from multiple sources can be simultaneouslydisplayed on a display of the display device. The information comingfrom different sources can either be displayed in separated parts of thedisplay or the different information can be displayed side by side oroverlaid together on the same part of the display.

Selecting a specific region on the display of the display device by forexample touching it can trigger different functions depending on thecircumstances. For example, a specific function can be activated ordeactivated, additional information can be displayed or a menu can beopened. The menu can offer the choice between different functions, forexample the possibility to adjust various parameter.

The adjustment of different parameters via a menu can be done in manyways, known from prior art and especially from the technology used inmobile phones with touch screen technology. Known are for examplescrolling or sliding gestures, swiping, panning, pinching, zooming,rotating, single, double or multi tapping, short or long pressing, withone or more than one finger of one or more hands and/or any combinationthereof.

A display device in combination with one or more cameras can be used toreplace a rearview mirror, either an interior or an exterior rearviewmirror. There are various advantages offered by this constellation. Forexample, can a display device together with a camera monitoring one sideof the vehicle and one camera monitoring the rear of the vehicle replacean external rearview mirror. By combining the pictures of both cameras,for example by the image improvement system as described before, theblind spot zone is eliminated and an improved visibility is offered.

The display devices can be arranged inside the vehicle eliminating theneed for exterior parts. This offers the advantage to smoothen the outershape of the vehicle, reduces the air friction and therefore offerspower and/or fuel savings.

The processing device can advantageously handle the input of multiplesources. Correlating the input data of the different sources allows forthe reduction of possible errors, increases measurement accuracy andallows to extract as much information as possible from the availabledata.

When driving it is especially important to perceive possibly dangeroussituations. One part of the processing device analyses the availabledata and uses different signaling means to enhance the perception of thesituation by the driver especially by the detection and classificationsystem described before. For example, can an object recognition andclassification algorithm be used to detect different objects surroundingthe vehicle, for example based on the pictures acquired by one or morecameras. Comparing the pictures for different points in time or usingsupplementary sensor data gives information about the relative movementof objects and their velocity. Therefore, objects can be classified intodifferent categories, for example dangerous, potentially dangerous,noted for continued observance, highly relevant, relevant, irrelevant.

From all the information, a level of danger attributed with each objectcan be derived. Depending on the danger level or other importantparameters, the perception of objects for the driver can be enhanced byusing different signaling means to display on the displaying device, forexample highlighting the objects with specific colors, increasedbrightness, flashing messages, warning signs and/or using audiomessages. The overall danger level or the highest danger level can alsobe displayed by special warning signs, like an increased brightness, acolorful border around the whole or specific parts of the display,constant in time or flashing with increasing or decreasing frequency.The information displayed on the display device is highly situationaland is reevaluated according to the updated information from the varioussensors and information sources. An emergency vehicle or a station canfor example broadcast an emergency message to allow for vehicles and thedriver of the vehicles for an improved reaction to possible dangeroussituations or to clear the path for emergency operations. A vehicleinvolved in an accident or dangerous situation can also broadcast amessage to call the attention of other vehicles and their drivers tothose situations.

A further adjustment possibility of the simulated exterior mirror is thefunction of adapting the field of view to the driver's position. As in aconventional mirror, which is adapted by an electric drive to theperspective of the driver, the mirror adjustment′ of the plane mirrorsimulation is done by moving section A on the optical sensor, so thatother pixels of the optical sensors are visualized. The number ofpixels, and therefore the size of the section, is not changed. Thisadjustment is indicated by the arrows in FIG. 9.

For a convex mirror, the adjustment to the perspective of the driver isnot connected with simply moving a pixel section, but rather with arecalculation of the image.

This modification might be restricted to modifications securing thatlegally prescribed view areas are still shown to the driver.Alternatively, the driver might receive a warning message in case themodification leads to a view area that is not fulfilling the legallyprescribed view areas.

In further embodiments, additionally and/or alternatively the displayedimage might be presented to the driver, especially in separated form, bya plurality of display devices. This allows to present specific detailsto the driver. Preferably also in this embodiment that driver is warnedin case the area displayed on a respective display device is notcovering a legally prescribed view area. Alternatively, is it possibleto always display the legally prescribed view area on a main displaydevice and display the specific details on a separate display deviceonly when necessary.

The whole control of the exterior mirror simulation is done by controlelements, which are used in the conventional way on the vehicle door oron the dashboard.

A rearview device is a type of device generally fulfilling the legalrequirements for rear view devices in vehicles, preferably motorvehicles. Therefore, such a rear-view device provides an image of therear area, substantially as provided for example in the state-of-the-artusing an interior mirror or a camera system and a display device, and ofthe lateral and backside areas lying diagonal to the sides,substantially as provided for example in the state-of-the-art usingexternal rearview mirrors or camera systems and display devices, theimage satisfying at least the legal requirements.

Such a rearview device within the subgroup of devices for indirect viewand as such the tasks, aims and the solutions described in thisinvention can also be used for indirect view devices. Examples areimages and views of objects which are not in the field of view of thedriver, i.e. the directions opposing, left, right, below and above ofthe viewing direction, but also the view along the direction of theviewing direction of the driver and/or any combinations of thedirections can be comprised. The view of the driver can be insufficientin particular also in the viewing direction, for example the view can beobstructed by vehicle parts of the vehicle itself, such as for exampleparts of the vehicle body, in particular the A-pillar, the roofconstruction and/or the engine hood, and the view can be obstructed byother vehicles and/or objects outside of the vehicle, obstructing theview in such a way that the driver can perceive the situation notcompletely satisfyingly or only incompletely. Additionally, it ispossible that the driver is not able to perceive the situation in orbeside the viewing direction in such a way to allow him to control thevehicle according to the present situation. Therefore, a rearview devicecan be adapted to reprocess the information according to the abilitiesof the driver to allow for a best possible perception of the situation.

It is also one aspect of the present invention to provide a rearviewdevice which can operate outside of the legal requirements and which canthen be admitted for operation either by exceptional permissions or anadaption of the legal requirements. Such a rearview device can beespecially inventiveness since it provides a solution outside of thepredefined well-known framework.

The rearview device can also be equipped with different illuminationdevices.

Different functions and devices can be incorporated into and/orcontrolled with the help of rearview devices, comprising especially alsocameras.

Especially useful are functions and devices to enhance, extend and/orsustain the functionality of the rearview device during normal orextreme conditions. This can comprise heating and/or cooling means,cleaning means such as wipers, liquid and/or gaseous sprays, actuatormeans for moving the rearview device or parts of it, such as for examplea display, a camera system and/or parts of a camera system, comprisingfor example lenses, filters, light sources, adaptive optics likedeformable mirrors, sensors and/or mirrors, and/or actuator means forinducing movement of other objects, for example parts of the vehicleand/or objects surrounding the vehicle. Furthermore, it can compriselinear tracks and/or rotating wheels, like for example a filter wheel,for exchanging optical elements, comprising for example lenses, mirrors,light sources, sensors, adaptive optics like deformable mirrors and/orfilters.

Prominent examples for functions and devices incorporated into and/orcontrolled with the help of rearview devices comprise also illuminationdevices, for example any kind of light module like an external lightmodule, an internal light module, a front light, a back light, a foglight, a brake light, an acceleration light, a turn signal, a logo lamp,a front area illumination light, a ground illumination light, a puddlelight, a flash light, a navigation light, a position light, an emergencylight, a spotlight, a green light, a red light, a warning light, a turnsignal light module, an approach light, a search light, an informationlight, a display and/or any combination thereof.

Further examples for functions and devices incorporated into and/orcontrolled with the help of rearview devices can comprise for example atiredness detection system, a microsleep detection system, a distanceand/or velocity determination system, for example a LIDAR (Lightdetection and ranging) system, a blind spot indicator system, a lanechange assistant system, a navigation assistant system, a trackingassistant system, a human-machine interaction system, a machine-machineinteraction system, an emergency and precaution assistant system, likean accident avoiding assistant system, a counter-measures assistantsystem, a brake assistant system, a steering assistant system, anacceleration assistant system, an escape assistant system, comprisingfor example an ejection seat system, a direction indicator, a blind spotindicator, an approach system, a strong braking system, an emergencybraking system, a charging status indicator, a vehicle mode system,comprising for example a sports mode system, an economy mode system, anautonomous drive mode system, a sleep mode system and an anti-theftsystem, a vehicle locked indicator system, a vehicle stolen indicator, awarning signal system, a temperature indicator system, a weatherindicator system, a traffic light signal system, a fuel status systemand/or any combination thereof.

A vehicle comprising at least two camera systems and processing meansare disclosed in German patent application No. 102012025322, filed onDec. 22, 2012 for MOTOR VEHICLE WITH CAMERA-DISPLAY-SYSTEM and herebyincorporated herein by reference. A process for the visualization of thesurrounding area around a vehicle is disclosed in German patentapplication No. 102012002149, filed on Feb. 12, 2012 for VISUALIZATIONPROCESS OF THE SURROUNDING OF A VEHICLE and hereby incorporated hereinby reference. A vision system for vehicles for displaying the image of arear-view mirror and of a wide-angle mirror is disclosed in Europeanpatent application No. 2623374, filed on Jan. 16, 2013 for VISION SYSTEMFOR UTILITY VEHICLES and hereby incorporated herein by reference.

A vehicle comprising an electronic rear view mirror is disclosed inEuropean patent No. 2822812, filed on Dec. 19, 2012 for MOTOR VEHICLEWITH AN ELECTRONIC MIRROR and hereby incorporated herein by reference. Alocking system adapted for unlocking in case of emergency is disclosedin German patent application No. 102012004792, filed on Mar. 7, 2012 forEMERGENCY UNLOCKING LOCKING SYSTEM and hereby incorporated herein byreference.

An example for a rearview device comprising an illumination devicefulfilling the brake light functions is disclosed in German patentapplication No. 102012108488, filed on Sep. 11, 2012 for REARVIEWASSEMBLY FOR MOTOR VEHICLE and hereby incorporated herein by reference.A light guidance unit for an illumination device used in a back-visionsystem is disclosed in German patent application No. 102012104529, filedon May 25, 2012 for LIGHT GUIDANCE UNIT which is hereby incorporatedherein by reference. An illumination device for a rearview device isdisclosed in German patent application No. 102012107833, filed on Aug.24, 2012 for ILLUMINATION DEVICE AND REARVIEW DEVICE which is herebyincorporated herein by reference. A lighting device for a back-visionunit is disclosed in German patent application No. 102012107834, filedon Aug. 24, 2012 for LIGHTING DEVICE AND BACK-VISION UNIT which ishereby incorporated herein by reference. A housing and display device ofa rearview device is disclosed in European patent No. 2738043, filed onDec. 3, 2012 for HOUSING AND DISPLAY DEVICE which is hereby incorporatedherein by reference. An optical light guide for a vehicle lighting unitis disclosed in European patent No. 2947378, filed on May 22, 2014 forOPTICAL LIGHT GUIDE FOR A VEHICLE LIGHTING UNIT which is herebyincorporated herein by reference. A display device of a rearview deviceof a vehicle is disclosed in International patent application No.2015/173695, filed on May 7, 2015 for DISPLAY DEVICE, REAR VIEW DEVICEAND MOTOR VEHICLE and claiming priority to European patent applicationNo. 2944866, filed on May 12, 2014 for OPTICAL UNIT, DISPLAY DEVICE,REAR VIEW DEVICE AND MOTOR VEHICLE INCLUDING THE SAME which are allhereby incorporated herein by reference. Further a light guiding devicefor an illumination device, in particular for a motor vehicle or adisplay device, in a rearview device of a motor vehicle is disclosed inEuropean patent application No. 3045944, filed on Jan. 19, 2015 forLIGHT GUIDING DEVICE which is hereby incorporated herein by reference.Still further a light guiding device for an illumination device,especially for a motor vehicle or an indicator device in a rearviewdevice of a motor vehicle is disclosed in U.S. patent application Ser.No. 15/228,566, filed on Aug. 4, 2016, for LIGHT GUIDING DEVICE and is acontinuation-in-part of U.S. patent application Ser. No. 15/000,733,filed on Jan. 19, 2016 for LIGHT GUIDING DEVICE which are all herebyincorporated herein by reference. In addition, an illumination device,particularly for a rear-view device of a motor vehicle and a method forproducing the same are disclosed in International patent application No.2016/147154, filed on Mar. 18, 2016 for ILLUMINATION DEVICE AND METHODFOR PRODUCING AN ILLUMINATION DEVICE and claiming priority to Germanpatent application No. 102015104163, filed on Mar. 19, 2015 forILLUMINATION DEVICE AND METHOD FOR PRODUCING AN ILLUMINATION DEVICEwhich are all hereby incorporated herein by reference. An improvedrear-view device for a motor vehicle which includes an electronic deviceis disclosed in U.S. patent application Ser. No. 15/256,532, filed onSep. 3, 2016 for ELECTRONIC DEVICE AND REAR-VIEW DEVICE and claimingpriority to European patent application No. 3139711, filed on Sep. 3,2015 for ELECTRONIC DEVICE AND REAR-VIEW DEVICE which are all herebyincorporated herein by reference. A lighting device for a rearviewdevice or a footwell device of a vehicle, comprising at least oneluminous means is disclosed in German patent application No.102015115555, filed on Sep. 9, 2015 for ILLUMINATION DEVICE, REAR VIEWDEVICE, FOOTWELL DEVICE AND VEHICLE which is hereby incorporated hereinby reference. A light module for a light assembly of an exteriorrear-view device is disclosed in European patent application No.3138734, filed on Sep. 3, 2015 for LIGHT MODULE, LIGHT ASSEMBLY AND REARVIEW DEVICE FOR A VEHICLE which is hereby incorporated herein byreference. A lighting device for a vehicle component, in particular fora rearview device of a motor vehicle, comprising a logo lamp and adeflection mirror are disclosed in European patent application No.3144183, filed on Sep. 13, 2016 for LIGHTING DEVICE, VEHICLE COMPONENTAND VEHICLE and claiming priority to German utility patent applicationNo. 202015104894, filed on Sep. 15, 2015 for LIGHTING DEVICE, VEHICLECOMPONENT AND VEHICLE which are all hereby incorporated herein byreference.

A camera module can comprise in particular a plurality of differentoptical elements, comprising a.o. a variety of sensors and lightsources, as well as housing parts.

The housing of a camera module can be made out of plastic, metal, glass,any other suitable material and/or any combinations thereof and can beused in combination with the techniques described below to change ormodify the properties of the material or the material surface. Housingsare for example described in German patent application No.102016108247.3, filed on May 3, 2016 for CLEANING SYSTEM FOR A CAMERAand U.S. patent application Ser. No. 15/281,780, filed Sep. 30, 2016 forTELESCOPING REARVIEW ASSEMBLY WITH CAMERA AND LENS WIPING SYSTEM, whichare all hereby incorporated herein by reference.

The camera can comprise for example CCD or CMOS or light field sensors,as for example described in German patent application No. 102011053999,filed Sep. 28, 2011 for DETECTION SYSTEM FOR OPTICAL DETECTION OF OBJECTAND/OR REGION OF SPACE FOR DRIVER ASSISTANCE AND/OR DISPLAY SYSTEMS OFMOTOR VEHICLE, HAS OPTICAL SENSOR ARRANGED AS LIGHT FIELD SENSOR FORDETECTION and U.S. patent application Ser. No. 09/771,140, filed on Jan.26, 2001 for MONITORING DEVICE FOR VEHICLES, IN PARTICULAR, MOTORVEHICLES, now U.S. Pat. No. 6,703,925, which are all hereby incorporatedherein by reference. Also, an area of the sensor can be reserved fordifferent purposes, for example to detect a test beam, as described inU.S. Pat. No. 8,031,224, filed on Sep. 9, 2014 for CAMERA SYSTEM, METHODFOR OPERATION OF A CAMERA SYSTEM AND SENSOR DEVICE OF A CAMERA SYSTEM,which is hereby incorporated herein by reference.

The optical elements can be molded or formed from any type of glass orany other suitable material. Glass is here used in the meaning of anon-crystalline amorphous solid showing a glass transition when heatedtowards the liquid state. It comprises for example the group ofpolymeric glasses, metallic glasses, silica glasses, but any othersuitable material showing the glass transition can also be used. Theglass can be either in a flat, wedge, rectangular, cylindrical,spherical, conical, elliptical, and/or circular shape, as described forexample in German patent application No. 102016108247.3, and Germanpatent application No. 102011103200, filed on May 31, 2011 for LIGHTWINDOW FOR USE AS LIGHT CONDUCTOR FOR TURN INDICATOR IN OUTSIDE MIRRORARRANGEMENT OF VEHICLE, HAS UNCOUPLING STRUCTURES AT CERTAIN LOCATION OFWINDOW, AND OPTICAL FILM WITH MOLDED COATING AND PROVIDED WITHUNCOUPLING STRUCTURES, which are all hereby incorporated herein byreference, or have a shape according to different needs or lens types.As non-limiting examples camera modules can be equipped with lenses,like a wide-angle or fish-eye lens suitable to provide peripheralimages, as described in U.S. patent application Ser. Nos. 15/281,780,and 13/090,127, filed on Apr. 19, 2011 for REAR VIEW MIRROR SIMULATION,now U.S. Pat. No. 9,238,434, a Fresnel lens or micro lenses as describedin German patent application No. 102011053999, filed Sep. 28, 2011 forDETECTION SYSTEM FOR OPTICAL DETECTION OF OBJECT AND/OR REGION OF SPACEFOR DRIVER ASSISTANCE AND/OR DISPLAY SYSTEMS OF MOTOR VEHICLE, HASOPTICAL SENSOR ARRANGED AS LIGHT FIELD SENSOR FOR DETECTION, and a TIR(total internal reflection) lens as described in U.S. Pat. No.8,740,427, filed Sep. 8, 2010 for OPTIMAL LIGHT COUPLING FOR REAR VIEWDEVICES, which are all hereby incorporated herein by reference. Anothertype of optical elements known to be used in camera modules are opticalfibers, especially in form of fiber bundles and preferably in form offiber bundles having an optical head, as described for example in U.S.patent application Ser. No. 09/771,140. Different methods can be used toproduce such optical elements, for example as described in U.S. Pat. No.8,460,060, filed on Jan. 30, 2009 for METHOD FOR CREATING A COMPLEXSURFACE ON A SUBSTRATE OF GLASS, which is hereby incorporated herein byreference.

The optical elements can be transparent as described for example in U.S.Pat. No. 8,031,224, German patent application No. 102016108247.3, andU.S. patent application Ser. No. 13/242,829, filed Sep. 23, 2011 forCAMERA ARRANGEMENT AND DOOR HANDLE FOR MOTOR VEHICLE, which are allhereby incorporated herein by reference. But the optical elements canalso be semitransparent, as described in U.S. patent application Ser.Nos. 09/771,140 and 13/090,127, which are all hereby incorporated hereinby reference. Still further, the optical elements can be completely orpartially coated with different type of coatings to realize differenteffects, such as for example anti-reflective coatings as described inU.S. Pat. No. 8,031,224, chromium-based reflective coatings as describedin U.S. Pat. No. 9,181,616, filed on Jan. 24, 2012 for CHROMIUM-BASEDREFLECTIVE COATING, and other coatings, for example for polymericsubstrates as described in U.S. patent application Ser. No. 14/936,024,filed on Nov. 9, 2015 for COATED POLYMERIC SUBSTRATES and in U.S. patentapplication Ser. No. 15/124,310, filed on Feb. 20, 2015 for DECORATIVECOATINGS FOR PLASTIC SUBSTRATES, which are all hereby incorporatedherein by reference. Preferably the optical elements are made of ascratch-proof material as described for example in German patentapplication No. 102016108247.3, which is hereby incorporated herein byreference. The optical elements can have uncoupling structures atcertain locations of the optical elements, and an optical film, forexample an extrusion film, and a molded coating can be applied asdescribed in German patent application No. 102011103200, which is herebyincorporated herein by reference. A coating to spectrally and stresscontrol is described in U.S. patent application Ser. No. 15/124,310,which is hereby incorporated herein by reference. Different filters canbe integrated into the optical elements such as for example gray filtersor polarization filters, described in U.S. patent application Ser. No.14/809,509, filed Jul. 27, 2015 for APPARATUS FOR LIGHT INTENSITYADJUSTMENT, which is hereby incorporated herein by reference.

Electrochromic substrates, polymer electrolytes and other chargeconducting medias may be used for the optical elements based on thedescriptions of European patent application No. 08103179.1, filed onMar. 31, 2008 for PROCESS FOR PRODUCING ELECTROCHROMIC SUBSTRATES ANDELECTROCHROMIC ARTICLES MADE THEREFROM, European patent No. 2202826,filed on Dec. 23, 2008 for POLYMER ELECTROLYTES AND DEVICES CONTAINING,U.S. Pat. No. 7,999,992, filed on Jan. 7, 2005 for CHARGE CONDUCTINGMEDIUM and U.S. Pat. No. 8,537,451, filed on Mar. 26, 2008 for PROCESSESFOR PRODUCING ELECTROCHROMIC SUBSTRATES AND ELECTROCHROMIC ARTICLES MADETHEREFROM, which are all hereby incorporated herein by reference.

The camera module can also be equipped with apparatuses for lightintensity adjustment as described for example in U.S. patent applicationSer. No. 14/809,509 and light level intensifier tubes as described inU.S. patent application Ser. No. 09/771,140, which are all herebyincorporated herein by reference. The electrochromic substrates anddevices used in European patent application No. 08103179.1, Europeanpatent No. 2202826, U.S. Pat. Nos. 7,999,992 and 8,537,451, which areall hereby incorporated herein by reference, can also be used for thispurpose as well as a transflector to transmit or reflect light based ona corresponding input signal, as described in German patent applicationNo. 102016106126.3, filed on Apr. 4, 2016 for IMAGING SYSTEM, which ishereby incorporated herein by reference.

The camera module or a cover adapted to the camera module can be movedusing different actuators, drives and/or a flexible track, as forexample described in German application No. 102016108247.3 and U.S.patent application Ser. No. 15/281,780, which are all herebyincorporated herein by reference.

Still further, the camera module can also comprise cleaning elements toclean the optical element facing outwards and being exposed to theenvironment. The cleaning element can for example comprise wipers,brushes, lips, nozzles, fans and similar elements as are described inEuropean patent application No. 14165197.6, filed Apr. 17, 2014 forOPTICAL SYSTEM FOR A VEHICLE, CLEANING DEVICE AND VEHICLE COMPRISING ANOPTICAL SYSTEM, U.S. patent application Ser. No. 15/281,780, Germanpatent application No. 102016108247.3, European patent application No.13163677.1, filed Apr. 15, 2013 for LENS WIPER, European patentapplication No. 15173201.3, filed Jun. 22, 2015 for LENS CLEANING WITHFLEXIBLE ACTUATOR and European patent No. 1673260, filed on Oct. 14,2003 for CLEANING DEVICE which are all hereby incorporated herein byreference. The cleaning devices are not limited in composition, and mayfor example comprise any fabric, elastomeric, sponge, brush, orcombination of these. Special wiper elements comprising wiper arms,wiper blades, wiping cloth, wiping tissue and combinations thereof aredescribed in European patent application No. 14165197.6, which is herebyincorporated herein by reference. A wiper element may for example becontrolled according to the method described in European patentapplication No. 130164250.6, filed Apr. 18, 2013 for METHOD FORCONTROLLING A WIPER DEVICE, which is hereby incorporated herein byreference. A reservoir for holding a cleaning liquid as described inEuropean patent application No. 14165197.6, which is hereby incorporatedherein by reference. Such a reservoir can be attached to or integratedinto the camera module to provide the cleaning liquid to the opticalelements of the camera module.

Different methods may be used to detect dirt or other obscurationspreventing or reducing the functioning of the camera module, such asdescribed in U.S. Pat. No. 8,395,514, filed on Jun. 24, 2008 for OPTICALSYSTEM AND METHOD FOR DETECTING OPTICAL SYSTEM OBSCURATION IN A VEHICLE,European patent No. 1328141, filed on January 12, for ASSEMBLY HAVING ACONDUCTOR FROM FLEXIBLE MATERIAL AND METHOD FOR MANUFACTURING SUCH ANASSEMBLY, and U.S. Pat. No. 8,031,224, which are all hereby incorporatedherein by reference.

Also, light sources can be installed or integrated into the cameramodule to increase the visibility of surrounding objects, measuredistances and directions and detect dirt, such as described in U.S. Pat.No. 8,031,224, U.S. patent application No. 62/470,658, filed on Mar. 13,2017, 2016 for LIGHT EMITTING MIRROR BEZEL and U.S. patent applicationSer. No. 09/771,140, which are all hereby incorporated herein byreference.

Different heating means, like heating coils, heating devices integratedinto the lens holder or the bezel, or other heating elements can be usedto impede condensation and icing at the surface of optical elements, asfor example described in German patent application No. 102016108247.3,U.S. patent application No. 62/470,658, and German patent applicationNo. 102016107545.0, filed on Apr. 22, 2016 for HEATING DEVICE FOR ACAMERA LENS, which are all hereby incorporated herein by reference.

A watertight seal against weather effects, as well as against theinfluence of washing processes with detergents, solvents andhigh-pressure cleaners can be used on the housing of the camera moduleas described in U.S. patent application Ser. No. 13/090,127, which ishereby incorporated herein by reference.

Alternatively, the housing can be made of a body comprising plastic andconductive material, wherein the conductive material is dispersed in theplastic material to form a conductive mass to allow a power source,preferably a DC voltage source, to connect via at least two electrodesto the body and heat the body accordingly, as described in German patentapplication No. 102016107545.0, which is hereby incorporated herein byreference.

A conductor track can be embedded within plastic parts of the cameramodule as described in European patent No. 1328141 and U.S. Pat. No.7,083,311, filed on Jan. 12, 2002 for CONDUCTOR OF FLEXIBLE MATERIAL,COMPONENT COMPRISING SUCH FLEXIBLE CONDUCTOR, AND METHOD OFMANUFACTURING SUCH CONDUCTOR, which are all hereby incorporated hereinby reference.

The camera module can comprise a power harvesting system as describedfor example in European patent application No. 09171683.7, filed on Sep.29, 2009 for SELF SUSTAINING REAR VIEW MIRROR, which is herebyincorporated herein by reference.

A fault detection system for electric consumers as described in U.S.Pat. No. 8,487,633 filed on Jan. 14, 2010 for FAULT DETECTION OFELECTRIC CONSUMERS IN MOTOR VEHICLES, which is hereby incorporatedherein by reference, can be used to detect failure of the camera module.

Different types of fixings can be used to fix the camera module to thevehicle or other components, such as for example the snap-fit connectiondescribed in European patent No. 2233360, filed on Mar. 27, 2009 forSNAP FIT CONNECTION IN A REAR VIEW MIRROR, which is hereby incorporatedherein by reference.

Different control means and analyzing devices can be used, such as thecomputation units described in U.S. patent application Ser. No.13/090,127, German patent application No. 102016106126.3, German patentapplication No. 102011053999, European patent No. 2146325, filed on July16, for Recording Device for Receiving, Processing and Storing ImageFiles in a Vehicle and Method, and U.S. Pat. No. 8,849,104, filed onJul. 16, 2008 for RECORDING DEVICE AND METHOD FOR CAPTURING ANDPROCESSING IMAGE DATA IN A VEHICLE, which are all hereby incorporatedherein by reference. In addition, HDR (high dynamical range) technologycan be used according to U.S. patent application Ser. No. 14/830,406,filed on Aug. 19, 2015 for REAR VIEW DEVICE FOR A MOTOR and published asUS 20150358590, which is hereby incorporated herein by reference.

The invention has been described in an illustrative manner. It is to beunderstood that the terminology, which has been used, is intended to bein the nature of words of description rather than of limitation.

Many modifications and variations of the invention are possible in lightof the above teachings. Therefore, within the scope of the appendedclaims, the invention may be practiced other than as specificallydescribed.

We claim:
 1. A method for displaying information graphically on an imagecaptured by an optical system used to simulate a rear-view mirror systemof a vehicle, the method comprising the steps of: capturing the image;identifying an object in the image; defining an entire outer peripheryof the object; assigning a priority level to the object based on apredetermined criterion; altering the object based on its priority levelto create an altered object by creating a halo disposed outside andadjacent the entire outer periphery such that the halo has a shapesimilar to the outer periphery of the object; and displaying the alteredobject in the image in place of the object to alert a vehicle operatorof its presence and the priority level associated therewith.
 2. A methodas set forth in claim 1 including the step of changing the prioritylevel as conditions between the object and the vehicle change.
 3. Amethod as set forth in claim 2 including the step of updating the haloaround the object due to changes in the priority level.
 4. A method asset forth in claim 3 wherein the step of updating the halo includeschanging one or more characteristic parameter of the halo, eachcharacteristic parameter being selected from a group including color,brightness, thickness, transparency, contrast and position of the halo.5. A method as set forth in claim 3 wherein the step of updating thehalo includes periodically switching the halo on and off to represent apulsating halo.
 6. A method as set forth in claim 1 including the stepof reverting the altered object back to the object when the prioritylevel of the object is reduced below a predetermined level.
 7. A methodas set forth in claim 1 wherein the step of identifying the object inthe image includes the step of characterizing the object as animate orinanimate prior to the step of creating the altered object.
 8. A methodas set forth in claim 1 wherein the step of identifying the object inthe image includes the step of characterizing the object as another roaduser prior to the step of creating the altered object.
 9. A method asset forth in claim 8 wherein the other road user is selected from agroup comprising a motor vehicle, a bicycle, a pedestrian and an animalprior to the step of creating the altered object.
 10. A method as setforth in claim 8 wherein the other road user is selected from a groupcomprising a car, a bus, a truck, a caravan, a police car, an ambulancevehicle prior to the step of creating the altered object.
 11. A methodas set forth in claim 7 or 8, wherein the object is altered to createthe altered object based upon the characterization of the object.
 12. Amethod as set forth in claim 1 wherein the step of altering the objectto create the altered object includes the step of coloring at least aportion of the object.
 13. A method as set forth in claim 1 wherein thestep of assigning a priority level to the object based on apredetermined criterion includes the step of assigning one of aplurality of priority levels to the object based on a set ofpredetermined criteria.
 14. A method as set forth in claim 13 whereineach of the plurality of priority levels relates to a separate degree ofalert.
 15. A method as set forth in claim 14 including the step ofchanging the altered object based on a set of external conditionswherein the set of external conditions are selected from a groupconsisting of ambient light, types of ambient light, temperature,precipitation and road conditions.
 16. A method as set forth in claim 14wherein the set of predetermined criteria are selected from a groupconsisting of relative distance, relative velocity, and relativeacceleration between the vehicle and the object.
 17. A method as setforth in claim 1 wherein the step of altering the object includes thestep of transforming pixels defining an area immediately adjacent theobject to create the halo.